Experiences with the Model-based Generation of Big Data Pipelines
نویسندگان
چکیده
Developing Big Data applications implies a lot of schematic or complex structural tasks, which can easily lead to implementation errors and incorrect analysis results. In this paper, we present a model-based approach that supports the automatic generation of code to handle these repetitive tasks, enabling data engineers to focus on the functional aspects without being distracted by technical issues. In order to identify a solution, we analyzed different Big Data stream-processing frameworks, extracted a common graph-based model for Big Data streaming applications and developed a tool to graphically design and generate such applications in a model-based fashion (in this work for Apache Storm). Here, we discuss the concepts of the approach, the tooling and, in particular, experiences with the approach based on feedback of our partners.
منابع مشابه
Security-Based Co-Expansion Planning of Gas and Electricity Networks Considering Demand Response
Coordinated expansion planning of power plants has always attracted the attention of power industry planners to arrive at the optimal mix that assures adequacy of energy supply. Here, we model the coordinates for expansion of electricity and gas networks while considering technical constraints, in order to minimize the total cost of investment and operation of electricity and gas networks for a...
متن کاملA model for predicting attitude of desire to change based on components of spirituality in the workplace and organizational commitment in the oil pipelines and telecommunications company of Fars district
The purpose of this research was to investigate the effects of spirituality in the workplace component of desire to change the attitude according to the mediating role of organizational commitment. This research based on its goal is considered a practical research which consumed a descriptive and correlative research based on data gathering. The oil pipelines and telecommunications company of F...
متن کاملGTrust: a group based trust model
Nowadays, the growth of virtual environments such as virtual organizations, social networks, and ubiquitous computing, has led to the adoption of trust concept. One of the methods of making trust in such environments is to use a long-term relationship with a trusted partner. The main problem of this kind of trust, which is based on personal experiences, is its limited domain. Moreover, both par...
متن کاملBig Data Analytics and Now-casting: A Comprehensive Model for Eventuality of Forecasting and Predictive Policies of Policy-making Institutions
The ability of now-casting and eventuality is the most crucial and vital achievement of big data analytics in the area of policy-making. To recognize the trends and to render a real image of the current condition and alarming immediate indicators, the significance and the specific positions of big data in policy-making are undeniable. Moreover, the requirement for policy-making institutions to ...
متن کاملGreen Energy Generation in Buildings: Grid-Tied Distributed Generation Systems (DGS) With Energy Storage Applications to Sustain the Smart Grid Transformation
The challenge of electricity distribution’s upgrade to incorporate new technologies is big, and electric utilities are mandated to work diligently on this agenda, thus making investments to ensure that current networks maintain their electricity supply commitments secure and reliable in face of disruptions and adverse environmental conditions from a variety of sources. The paper presents a new ...
متن کامل